Nonmonotone and Perturbed Optimization

The primary purpose of this research is the analysis of nonmonotone optimization algorithms to which standard convergence analysis techniques do not apply. We consider methods that are inherently nonmonotone, as well as nonmono-tonicity induced by data perturbations or inexact subproblem solution. One of the principal applications of our results is the analysis of gradient-type methods that process the data incrementally. The computational signiicance of these algorithms is well documented in the neural networks literature. Such algorithms are known to be particularly well-suited for large data sets, as well as for real-time applications. One of the most important methods of this type is the classical online backpropagation (BP) algorithm for training artiicial neural networks. Neural networks constitute a large interdisciplinary area of research within the broader area of machine learning that has found applications in many branches of science and technology. However, much of the work in the area has been based on heuristic concepts and trial-and-error experimentation. This research lls some of the existing theoretical gaps. In particular, we obtain the rst deterministic convergence results for the BP algorithm and its various ii practically important modiications. We also investigate error-stability properties of the generalized gradient projection method. When specialized to neural network training, our general results allow us to establish stability of BP in the presence of noise, and give its precise characterization. We also outline applications to weight perturbation training. In a classical optimization setting, some new results are derived for a perturbed generalized gradient projection method applied to convex and weakly sharp problems. Next we develop a general approach to convergence analysis of feasible descent methods in the presence of perturbations. The important novel feature of our analysis is that perturbations need not tend to zero in the limit. In this case, standard convergence analysis techniques are not applicable, and we present a new approach. It is shown that a certain "-approximate solution can be obtained , where " depends on the level of perturbations linearly. Applications to the gradient projection, proximal minimization and extragradient algorithms are described. We also consider a practical generalization of the parallel variable distribution algorithm of Ferris and Mangasarian. In particular, our generalization is twofold : we propose an asynchronous algorithm, and allow inexact subproblem solution. We show that the generalized method retains all the attractive properties of the original method and yet is more practical. We also derive some stronger convergence results …